Evaluation of Open- vs. Restricted- Domain Question Answering Systems in the Biomedical Field

Question Answering Systems (hereinafter QA systems) stand as a new alternative for Information Retrieval Systems. We conducted a study to evaluate the efficiency of QA systems as terminological sources for physicians, specialized translators, and users in general. To this end we analysed the performance of two open-domain and two restricteddomain QA systems. The research entailed a collection of one hundred fifty definitional questions from WebMed. We studied the sources that QA systems used to retrieve the answers, and later applied a range of evaluation measures to mark the quality of answers. Through analysing the results obtained by asking the 150 questions in the QA systems MedQA, START, QuALiM and HONqa, it was possible to evaluate the systems’ operation through applying specific metrics. Despite the limitations demonstrated by these systems, as they are not accessible to everyone and they are not always completely developed, it has been confirmed that these four QA systems are valid and useful for obtaining definitional medical information in that they offer coherent and precise answers. The results are encouraging because they present this type of tool as a new possibility for gathering precise, reliable and specific information in a short period of time. 1 Correspondence to: Juncal Gutiérrez-Artacho, Facultad de Traducción e Interpretación, C/ Buensuceso 11 s/n, Universidad de Granada, [email protected] Olvera-Lobo, María-Dolores & Gutierrez-Artacho, Juncal Journal of Information Science, XX (X) 2009, pp. 1–11 DOI: 10.1177/0165551506nnnnnn 2 © The Author(s), 2009, Reprints and Permissions: http://www.sagepub.co.uk/journalsPermissions.nav